General Medicine 



OPEN ACCESS 



Research Article 



Development and validation of a generic questionnaire 
for the implementation of complex medical interventions 

Entwicklung und Validierung eines generischen Fragebogens fur die 
Implementierung komplexer medizinischer Interventionen 



Abstract 

Introduction: The implementation of complex medical interventions in 
daily practice is often fraugint witii difficulties. Accordingto the iterative 
phase model proposed by the British Medical Research Council (MRC), 
the development, implementation and evaluation of complex interven- 
tions should be theory-driven. A conceptual model that seems to be a 
promising framework is the Theory of planned behaviour (TPB). In our 
study we aimed to develop and validate a generic and multifaceted 
questionnaire based on the TPB to detect physicians' willingness to 
implement complex medical interventions and the factors influencing 
this willingness. 

Methods: The questionnaire was developed according to the literature 
and was informed by previous qualitative research of our department. 
It was validated on the example of an electronic library of decision aids, 
arriba-lib. The sample consisted of 181 General Practitioners (GPs) who 
received a training regarding arriba-lib and subsequently filled in the 
questionnaire, assessing the TPB variables attitude, subjective norm, 
perceived behaviour control and intention. Follow-up assessments were 
conducted after two (assessing retest reliability) and eight weeks (as- 
sessingtarget behaviour). We performed a confirmatory factor analysis 
investigating the factorial structure of our questionnaire according to 
the TPB. Beside the calculation of the questionnaire's psychometric 
properties we conducted a structural equation model and an ordinal 
regression to predict actual behaviour regarding the installation and 
application of arriba-lib. 

Results: The postulated three factorial model (attitude, subjective norm, 
perceived behaviour control) of our questionnaire based on the TPB 
was rejected. A two factorial model with a combined factor subjective 
norm/perceived behaviour control was accepted. The explained variance 
in the ordinal regression was low (Nagelkerke's R^=.12). Neither attitude 
nor intention were able to predict the use or non-use of arriba-lib (atti- 
tude: p=.68, intention: p=.44). For the combined factor subjective 
norm/perceived behaviour control a significant, but small effect (p=.03) 
was shown. 

Conclusions: The TPB is not an adequate theoretical framework to guide 
the development of a generic questionnaire in the context of the imple- 
mentation of complex interventions. To enable the successful imple- 
mentation of complex medical interventions evaluators have to go 
through the whole development and evaluation process according to 
the MRC-model, without short cuts. Further, it has to be discussed if a 
generic instrument can be valid and useful. Regarding the TPB a publi- 
cation bias regarding the theory's applicability might have to be con- 
sidered. 

Keywords: implementation, questionnaires, primary health care, theory 
of planned behaviour 
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Zusammenfassung 

Einfiihrung: Die Implementierung komplexer medizinischer Interventio- 
nen in dietagliche Praxis istoftmals mitSchwierigl<eiten erfullt. GemaS 
dem schrittweisen Phasenmodell des British IVIedical Research Council 
(MRC) sollten die Entwicl<lung, Implementierung und Evaluation kom- 
plexer medizinischer Interventionen theoriegeleitet sein. Die Theorie 
des geplanten Verhaltens (TGV) kdnnte als konzeptuelles Modell einen 
entsprechenden Rahmen darstellen. In unserer Studie beabsichtigten 
wir die Entwicklung und Validierung eines generischen und facettenrei- 
chen Fragebogens basierend auf der TGV, um die Bereitschaft von 
Medizinern, komplexe medizinische Interventionen zu implementieren 
zu erheben sowie die Faktoren, die diese Bereitschaft beeinflussen. 
Methoden: Unser Fragebogen wurde auf der Basis der Literatur und 
vorhergehender qualitativer Forschung unserer Abteilung entwickelt. 
Er wurde validiert am Beispiel einer elektronischen Bibliothek von Ent- 
scheidungshilfen, arriba-lib. DieStichprobe bestand aus 181 Allgemein- 
medizinern, die ein Training hinsichtlich arriba-lib erhielten und anschlie- 
Send den Fragebogen ausfullten, wobei sie die TGV-Variablen Einstel- 
lung, subjektive Norm, wahrgenommene Verhaltenskontrolle und Inten- 
tion beurteilten. Follow-up Untersuchungen erfolgten nach zwei (Retest- 
reliabilitat) und acht Wochen (Erfassung des Zielverhaltens). Wir rech- 
neten eine konfirmatorische Faktorenanalyse, um die faktorielle 
Struktur unseres Fragebogens hinsichtlich der TGV zu untersuchen. 
Neben der Berechnung der psychometrischen Eigenschaften des Fra- 
gebogens fuhrten wir ein Strukturgleichungsmodell und eine ordinale 
Regression durch um das tatsachliche Verhalten der Installation und 
Anwendung von arriba-lib vorher zu sagen. 

Ergebnisse: Die postulierte dreifaktorielle Struktur (Einstellung, subjek- 
tive Norm, wahrgenommene Verhaltenskontrolle) unseres Fragebogens 
basierend auf der TGV wurde abgelehnt. Ein zweifaktorielles Modell mit 
einem kombinierten Faktor subjektive Norm/wahrgenommene Verhal- 
tenskontrolle wurde akzeptiert. Die erklarte Varianz im Rahmen der 
ordinalen Regression war gering (Nagelkerke's R^=.12). Weder Einstel- 
lung noch Intention konnten die Verwendung oder Nichtverwendung 
von arriba-lib vorhersagen (attitude: p=.68, intention: p=.44). Fur den 
kombinierten Faktor subjektive Norm/wahrgenommene Verhaltenskon- 
trolle zeigte sich ein signifikanter, aber kleiner Effekt (p=.03). 
Schlussfolgerung: Die TGV ist kein angemessenertheoretischer Rahmen 
fur die Entwicklung eines generischen Fragebogens im Kontext der 
Implementierung komplexer Interventionen. Um eine erfolgreiche Imple- 
mentierung komplexer medizinischer Interventionen zu erreichen, muss 
der komplette Entwicklungs- und Evaluationsprozess des MRC-Modells 
bearbeitet werden. Des Weiteren sollte diskutiert werden, ob ein gene- 
risches Instrument valide und nutzlich sein kann. Hinsichtlich der TGV 
kdnnte ein Publikationsbias bezuglich der praktischen Anwendung der 
Theorie vorliegen. 

Schliisselworter: Implementierung, Fragebdgen, Allgemeinmedizin, 
Theorie geplanten Verhaltens 



Introduction 

The implementation of complex medical interventions to 
improve the quality of care is fraught with difficulties as 
the intervention is often not taken up by the target group 
[1]. Thereby, complex interventions like guidelines, 
treatment pathways or decision aids are defined as inter- 
ventions with several interrelated components [2]. As 



other research findings, the implementation of complex 
interventions face the gap between what is known to be 
best practice and the care patients actually receive [3], 
[4], [5]. For research results to be transferred into regular 
healthcare delivery it may take 10 to 20 years [6]. Mul- 
tiple factors may influence physicians' professional beha- 
viour change and thus the lasting implementation of 
complex medical interventions. These factors are situated 
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Development 

1 Identifying the evidence base 

2 Identifying/developing theory 

3 Modelling process and outcomes 



Feasibility/piloting 

1 Testing procedures 

2 Estimating recruitment /retention 

3 Determining sample size 




Evaluation 

1 Assessing effectiveness 

2 Understanding change process 
3. Assessing cost-effectiveness 



implementation 

1 Dissemination 

2 Surveillance and monitoring 

3 Long term follow-up 



Figure 1: Key elements of the development and evaluation process, redrawn witli permission according to [2] 



Attitude 



Subjective Norm 



Intention 



Behaviour 



Perceived Behaviour 
Control 



Figure 2: Theory of planned behaviour, based on [8] 



at a number of levels (individual, colleagues, patients, 
system) [7]. As the implementation and evaluation of 
complex interventions requires time, financial and human 
resources it would be highly desirable to develop and 
provide a standardized instrument to measure health 
care professionals' willingness to implement an innova- 
tion. Thus by identifying physicians' willingness to imple- 
ment complex medical interventions and the factors influ- 
encing this willingness possible conclusions (regarding 
the abortion or modification of the implementation pro- 
cess) could be drawn at an early stage. According to the 
iterative phase model (2) (see Figure 1) proposed by the 
British Medical Research Council (MRC) the development, 
implementation and evaluation of complex medical inter- 
ventions should be theory-driven. 
The use of conceptual models is considered to facilitate 
the choice for an adequate implementation strategy and 
thus make the implementation more likely. Theories ex- 
plaining the changes that occur by implementation are 
numerous and focus on different levels (individual, inter- 
personal, and organizational). In this context, an estab- 
lished theory that offers a promising framework for 
studying the factors associated with physicians behaviour 
is the Theory of planned behaviour (TPB) [8], which is one 
of the best validated psychological models to predict in- 



dividual behaviour [9], [10]. The theory shown in Figure 2, 
assesses a persons' intention to perform a specific beha- 
viour as a proximal predictor of behaviour. 
According to the theory three kinds of considerations 
impact on a person's behaviour: behavioural (about the 
outcome of performing the behaviour), normative (about 
the expectations of other people to act) and control beliefs 
(about factors that facilitate or hinder the performance 
of the behaviour). Their respective cognitive aggregates 
(behavioural beliefs are associated with attitude, norma- 
tive beliefs with subjective norm and control beliefs with 
perceived behaviour control) are assumed to influence 
intention to perform the behaviour, while the intention 
itself is considered as the main predictor for future beha- 
viour. Attitude means a person's overall evaluation of the 
behaviour while subjective norm describes a person's 
own estimate of the social pressure to perform or not 
perform the behaviour under consideration. The third 
component perceived behaviour control is defined as the 
extent to which a person feels able to enact the beha- 
viour. The perceived behaviour control consists of two 
subfactors: how much a person has control over the be- 
haviour and how confident a person feels about being 
able to perform or not perform the behaviour. Ajzen con- 
sidered the concept of perceived behaviour control as 
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being congruent to the dimension of self-efficacy, pro- 
posed by Bandura [11]. Beside its indirect impact on the 
behaviour, mediated by the intention, perceived behaviour 
control also has a direct effect on behaviour. While in 
previous health care studies the TPB was used to explain 
behavioural change on patients' side (i.e. condom use 
[12], physical activity [13]) in recent years the behavioural 
change of health care professionals came to the fore- 
ground [14], [15], [16] and has shown its suitability in a 
variety of medical consultations [17], [18], [19]. Despite 
its wide distribution and the fact, that the TPB explains 
almost one third of variance in the behaviour of health 
care professionals as suggested by a systematic review 
[20], the assumptions of theTPB regarding the predictive 
value of intention for actual behaviour are discussed 
controversially [17], [21]. 

As mentioned above, the lasting implementation of 
complex interventions is difficult to achieve and factors 
influencing physicians' willingness to take up the interven- 
tion are often identified late in the implementation pro- 
cess. Therefore, we aimed in our study to develop and 
validate a generic and multifaced questionnaire based 
on the TPB to detect physicians' willingness to implement 
complex medical interventions and the factors influencing 
this willingness. 

Methods 

Development of the questionnaire 

The questionnaire's development and validation consisted 
of two phases: At first we aimed to design the question- 
naire referring to a concrete complex medical intervention 
of potential interest for the participating General Practi- 
tioners (GPs). In a second step it was planned to transfer 
and cross-validate the research findings in further com- 
plex interventions to guarantee its generic character. In 
this article we describe the first development phase using 
the example of arriba-lib, a multimodular electronic library 
of decision aids [22], [23]. 

The questionnaire was developed in view of the literature 
on the construction of a TPB based questionnaire [24], 
[25]. As recommended [24], [26] we used qualitative in- 
terviews to elicit salient beliefs of GPs about facilitators 
and barriers of adheringto complex interventions. These 
were derived from previous qualitative research of our 
department on the implementation of different complex 
interventions [7], [27]. According to the principle of 
compatibility [28] the four TPB predictors referred to the 
use of arriba-lib in general and not to one specific module. 
We pretested the initial draft with 5 GPs of the depart- 
ment's GP network, which were not included in the final 
study. Minor revisions of wording were made to the 
questionnaire in the light of their comments. To enable 
the adaptation of the questionnaire to other complex in- 
terventions we only used items whose content and 
wording could be transferred to other contexts. 



As we considered all areas elicited in the qualitative re- 
search the final version of the questionnaire (see Attach- 
ment 1) included 41 items and incorporated a different 
item number for each of the TPB variables, namely atti- 
tude (26 items), subjective norm (8 items) and perceived 
behaviour control (5 items). Thus, as recommended, each 
variable was assessed by multiple items to increase 
measurement reliability [26]. The amount of 26 items for 
attitude derives from the assumption that attitude is a 
more complex, multifaceted construct than subjective 
norm and perceived behaviour control. Intention was 
measured by two items ("In half a year arriba-lib will be 
routine in my practice."; "I intend to use arriba-lib in every 
consultation, if a corresponding problem exists."), which 
were summed up to predict the target behaviour, opera- 
tionalized as the installation and application of arriba-lib. 
After transforming reversed items, higher values of scales 
indicated stronger/more positive influences, attitudes 
and intention to install and apply arriba-lib. Additionally, 
the questionnaire included questions about demographic 
details and the practices. 

Due to practical considerations we deviated in some 
points from the recommendations [25]. In contrast to 
most TPB studies using a seven-point scale with verbal 
expressions of endpoints, we used a four-point Likert 
Scale with wording (1 = I do not agree, 2 = I rather do not 
agree, 3 = I rather agree, 4 = I fully agree) as response 
category. Further, as adding indirect measures is not 
considered to increase the level of prediction [29], we 
assessed the TPB variables directly by asking the parti- 
cipants to summarize their overall attitude, subjective 
norm, perceived behaviour control. In contrast to the in- 
direct assessment of the TPB variables where specific 
beliefs and scores for the single variables are measured 
separately and then combined by the researcher, the 
direct assessment considers the respondents to combine 
both aspects themselves. 

Study design and sample 

Our survey study ran from July 2011 to March 2012. The 
target population included interested GPs in Germany, 
who were contacted via different GPs university networks. 
At baseline GPs received a standardized group training 
where arriba-lib was presented. Arriba-lib [30] presently 
contains evidence-based decision aids for the following 
topics: cardiovascular prevention, atrial fibrillation, 
coronary heart disease, dual platelet inhibition, oral anti- 
diabetics, conventional and intensified insulin therapy, 
and unipolar depression. The modules are structured to 
assist physicians in counselling their patients according 
to the philosophy of shared decision making (SDM) [31]. 
An evaluation study showed that arriba-lib was accepted 
by a high number of patients and was also positively 
evaluated by GPs [22], [30]. Subsequent to the training 
a convenience sample of GPs filled in the questionnaire 
and gave their written informed consent to be contacted 
forthe follow-up assessments. Confidentiality for the GPs 
was assured by a pseudonymous procedure. Ethic approv- 



eiournal 



GMS German Medical Science 2014, Vol. 12, ISSN 1612-3174 



4/11 



Kramer et al.: Development and validation of a generic questionnaire 



al for the study was obtained from tine Ettnics Committee 
of the Facuity of IVledicine at ttie Ptiiiiips University of 
IVlarburg, Germany. In return for tiieir study participation, 
the GPs received a free one year licence for the arriba-lib 
software (beta version). To assess retest reliability, GPs 
were two weeks later reminded by mail to fill in again the 
questionnaire and to indicate if they installed and/or used 
the arriba-lib software. 

Further 6 weeks later GPs were contacted via phone to 
report on their (non-) use of the software and their evalu- 
ation of arriba-lib as a whole and its single modules. 

Data analysis 

We analyzed baseline characteristics of the GPs to 
identify possible selection bias between study dropouts 
and participants. Cross-tab analyses with x^-tests and 
standardized residuals were performed using Cramer-V 
as an effect size. A value of .40 or higher denotes a large 
effect [32], [33]. Mann-Whitney U tests were calculated 
to examine absolute differences in metric variables. 
In order to examine the postulated three factorial struc- 
ture (attitude, subjective norm, perceived behaviour 
control) of the questionnaire according to the TPB, we 
conducted a confirmatory factor analysis with unweighted 
least squares (ULS) as this estimation method makes no 
distributional assumptions [34] . Several fit indices were 
used to evaluate the factor analytic solution. The root 
mean square residual (RMR) measures the mean abso- 
lute value of the covariance residuals [35]. Values less 
than .05 indicate a good model fit [34], but other authors 
state that a value of less than .10 signals an acceptable 
model fit [36], [37]. The standardised root mean square 
residual (SRMR) eliminates scaling effects of the RMR. 
Values < .10 indicate a good model fit [37]. The Global 
Fit Index (GFI) can be considered as a measure of the 
proportion of variance and covariance that a given model 
is able to explain. A GFI equal or higher .90 can be con- 
sidered as reflecting a good model fit [38]. The adjusted 
global fit index (AGFI) takes the number of parameters 
used in computing the GFI into account. An AGFI equal 
or higher .90 can be considered as showing a good 
model fit [37]. 

Internal consistencies of the factors were examined by 
Cronbach's a coefficients. Values >.7 are considered to 
be acceptable [39]. Spearman's correlation coefficients 
were used to examine retest reliability. Values >.7 are 
considered to be acceptable. Differences in absolute 
values between the two administrations of the question- 
naire were tested by the Wilcoxon Matched Pairs Signed 
Rank Test with an alpha level <.05. We tested a structural 
equation model representing the TPB with unweighted 
least squares (ULS) and the above mentioned fit indices. 
Furthermore, due to the ordinal scale level of the outcome 
behaviour, an ordinal regression was used to predict ac- 
tual behaviour regarding arriba-lib ("not installed, not 
applied", "installed, not applied", "installed, applied") 
assessed in the telephone interview by components of 



the theory of planned behaviour operationalized in our 
questionnaire at baseline [40]. 
Missing values were replaced using the expectation 
maximization (EM) algorithm [41]. All calculations were 
done using SPSS 19.0 and AMOS 19.0. 

Results 

Sample characteristics 

At baseline a total of 181 GPs participated in the study. 
As seen in Table 1 the majority of the participating physi- 
cians were male (59%) and full-time working (90%). They 
were on average 51.9 years old (SD=7.6) The number of 
181 GPs participating in our research can be considered 
as adequate for the performed analyses [37]. Additionally, 
our sample can be regarded as representative as demo- 
graphic characteristics of primary care physicians in 
Germany are very similar to those assessed in our study 
[42]. 

To the first follow-up assessment measuring retest reli- 
ability 46.6% of the original sample (n=84) participated 
in the postal survey. At the second follow-up, where GPs 
were called by phone, the drop-out rate due to refusal 
accounted for 9.9% (n=18) compared to baseline. 
Figure 3 depicts the flow of the sample. There were no 
differences between the final sample and dropouts in 
sociodemographic characteristics at first follow-up, except 
for working hours in which the two groups differed signi- 
ficantly as the participants at the first follow-up were more 
often working part-time (p=.02), but the resulting effect 
size was low (Cramer-V: .17). 



n=181 
baseline 






n=84 

first follow up 
2 weeks 
postal survey 








n=163 

second follow up 
8 weeks 

telephone survey 





Figure 3: Flow of the sample 
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Table 1: Sociodemographic and baseline characteristics of the study population (n=181) 



to 



Demographics and professional characteristics 


n 


(%)" 


Gender 






Male 


107 


(59) 


Female 


74 


(41) 


Age (years) 






<40 


11 


(6) 


41 to 50 


64 


(35) 


>51 


106 


(59) 


Established since (years) 






<5 


g 


(10) 


6 to 15 


32 


(33) 


16 to 25 


35 


(37) 


>26 


19 


(20) 


Characteristic of the practice 






Single practice 


80 


(44) 


Group practice 


97 


(54) 


Patients per 3 months period 






<500 


5 


(3) 


500-1 ,000 


45 


(25) 


1,000-1,500 


62 


(34) 


>1,500 


65 


(36) 


Practice location (regarding inhabitants) 






<5,000 


36 


(20) 


5,000 to 20,000 


48 


(27) 


20,000 to 100,000 


25 


(14) 


>1 00,000 


69 


(38) 


Status 






Full time practice 


162 


(90) 


Part time practice 


18 


(10) 


^ Numbers may not add up to 181 and percentages may not add up to 100% due to 



missing values and rounding 



Confirmatory Factor Analysis (CFA) 

Descriptive statistics (mean, standard deviation) for each 
item at baseline were shown in Table 2. 
There was a maximum of 5% missing values on single 
variables (tO) which were missing completely at random 
(Little's MCAR-Test p=.19). These were imputed using 
the expectation maximization (EM) algorithm [41]. 



The three factorial model was rejected because of a not 
positive definite covariance matrix between the three 
postulated factors (see Attachment 2). Results suggested 
to combine subjective norm and perceived behaviour 
control in one factor. This two factorial model had an ac- 
ceptable model fit (SRMR=.08, RMR=.04, GFI=.92, 
AGFI=.92) although the correlation between the combined 
factor and factor "attitudes" was .90. Table 3 depicts the 
factor loadings of the items on their respective factors. 
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Table 2: Descriptive statistics of tiie questionnaire's items at baseline 



Item 


Mean* 


SD 


Postulated factors 


1. When 1 use arriba-lib, 1 have the impression to do something positive for my 
patients. 


3.3 


.62 


Attitude 


2. 1 am expected to use arriba-lib. 


1.8 


.80 


Subjective Norm 


3. It depends on myself if 1 use arriba-lib in the consultation. 


3.6 


.69 


Perceived behaviour 
control 


4. 1 can identify myself with the developer group of arriba-lib. 


3.0 


.74 


Attitude 


5. 1 think that arriba-lib will be accepted well by my patients. 


3.1 


.60 


Subjective norm 


6. arriba-lib helps me to structure the treatment of my patients. 


3.1 


.59 


Attitude 


7. In my opinion the design (layout, choice of colour, clarity) of arriba-lib is 
accurate. 


3.1 


.60 


Attitude 


8. 1 am confident to use arriba-lib in the consultation, if 1 like to. 


3.3 


.63 


Perceived behaviour 
control 


9. IVIy patients would appreciate if 1 use arriba-lib. 


3.1 


.56 


Subjective norm 


10. arriba-lib fits in the current health care system. 


2.9 


.78 


Attitude 


1 1 . In half a year arriba-lib will be routine in my practice. 


2.6 


.78 


Intention 


12. In my opinion arriba-lib is evidence-based. 


3.4 


.59 


Attitude 


13. 1 think that 1 can change in that way, that 1 can use arriba-lib in my daily 


3.0 


.63 


Perceived behaviour 


practice. 






control 


14. 1 worry that the use of arriba-lib can lead to drug regresses, (i) 


3.5 


.63 


Attitude 


15. 1 think that my GP colleagues will accept arriba-lib positively. 


2.8 


.61 


Subjective norm 


16. arriba-lib helps me to make decisions together with my patients. 


3.4 


.61 


Attitude 


17. 1 am open-minded towards medical innovations like decision aids or 


3.4 


.61 


Attitude 


guidelines. 








18. For me arriba-lib is well comprehensible. 


3.3 


.59 


Attitude 


19. Colleagues that are important to me think that 1 should use arriba-lib. 


2.5 


.84 


Subjective norm 


20. In the past 1 had already positive experiences with medical innovations like 
decision aids or guidelines. 


3.0 


.89 


Attitude 


21 . By arriba-lib 1 feel restricted in my freedom of choice, (i) 


1.5 


.68 


Attitude 


22. My evaluation of arriba-lib depends on whether 1 was involved in the 
development of arriba-lib. 


3.5 


.68 


Attitude 


23. For me the use of arriba-lib is difficult, (i) 


3.3 


.78 


Perceived behaviour 
control 


: : : — —. : 

24. 1 worry that the use of arriba-lib can have judicial consequences for me. (i) 


Q T 
O. / 


.OD 


Attituue 


25. For me arriba-lib is plausible. 


3.3 


.58 


Attitude 


26. The use of arriba-lib means a high temporal effort for me. (i) 


2.3 


.74 


Attitude 


27. By the use of arriba-lib 1 feel safer in my practice. 


3.0 


.75 


Attitude 


28. arriba-lib facilitates the work with my patients. 


3.0 


.64 


Attitude 


29. 1 like to test medical innovations like decision aids or guidelines. 


3.2 


.75 


Attitude 


30. 1 already treated my patients like arriba-lib recommends, (i) 


2.1 


.68 


Attitude 


31. arriba-lib should be embedded in other programs like DMP or GP 


3.0 


.86 


Attitude 


contracts. 








32. 1 feel under social pressure to use arriba-lib. 


1.5 


.63 


Subjective norm 


33. The use of arriba-lib reduces the uncertainty in treatment. 


2.9 


.74 


Attitude 


34. In the past 1 had already negative experiences with medical innovations like 
decision aids or guidelines, (i) 


3.4 


.75 


Attitude 


35. 1 intend to use arriba-lib in every consultation, if a corresponding problem 
exists. 


2.4 


.80 


Intention 


36. 1 find the implementation of arriba-lib in Germany desirable. 


3.0 


.68 


Attitude 


37. arriba-lib can be integrated well in daily practice. 


2.8 


.66 


Attitude 


38. The use of arriba-lib gives me the security to do the right. 


2.9 


.72 


Attitude 


39. arriba-lib means a significant change of my treatment routine, (i) 


2.8 


80 


Attitude 


40. 1 am often sceptic towards new innovations like decision aids or 


3.0 


.88 


Attitude 


guidelines, (i) 








41 . A regular support for the use of arriba-lib in form of information material, 


2.7 


.80 


Attitude 


trainings, phone hotline etc. is desirable. 









i = inverted 

* = Likert-scale from 1 = I do not agree to 4 = I fully agree 
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Table 3: Factor loadings in tlie two factorial model 



factor loading 


dLLI LUUco 




itsm 1 




it6rn 4 


RR 
.DO 


itom R 

IICI 1 1 \J 


R/1 


itsm 7 


.o^ 








.OO 


ILcllI lO 


Rft 
.OO 


itsm 1 7 


.O 1 


llclll lO 


.OO 


itom on 
ILcllI 


^iR 
.OO 
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Items 10, 22, 26, 30, and 39 have loadings <|.40| on 
factor "attitudes" and items 2, 3, 14, 15, 24, and 32 have 
loadings < | .40 1 on the combined factor "subjective norm 
and perceived behaviour control". 

Psychometric properties of the 
questionnaire 

Internal consistency coefficients and retest reliability for 
factor "attitude" were acceptable at both measuring 
points (Internal consistency coefficients: tO: a=.87, tl: 
a=.90; retest reliability: .75), whereas the values for the 
combined factor subjective norm and perceived behaviour 



control were not satisfying (Internal consistency coeffi- 
cients: tO: 0(=.65, tl: o(=.60; retest reliability: .41). The 
median differences of absolute values between baseline 
and retest did not differ significantly from 0 (attitude: 
p=.15,Z=-1.43; subjective norm and perceived behaviour 
control: p=.54, Z=-.61; Wllcoxon Matched Pairs Signed 
Rank Test) 

Predicition of actual behaviour 

Regardlngthe target behaviour 64 GPs (39.9%) Indicated 
to have neither Installed nor applied arrlba-llb In the past 
eight weeks. 26 physicians (16%) Installed arrlba-llb but 
did not use the software. Most of the participants (n=73, 
44.8%) Installed and applied arrlba-llb. 
A structural equation model to predict actual behaviour 
regarding arrlba-llb assessed In the telephone Interview 
by components of our modified theory of planned beha- 
viour operatlonallzed In our questionnaire at baseline 
was rejected. We therefore calculated an ordinal regres- 
sion with actual behaviour regarding arrlba-llb ("not In- 
stalled, not applied", "Installed, not applied", "Installed, 
applied") as dependent variable and sum scores of factors 
"attitudes" and "subjective norm/perceived behaviour 
control" and "Intention", operatlonallzed as sum score 
of Items 11 and 35, as predictors. The explained variance 
of the model was low (Nagelkerke's R^=.12). The test of 
parallel lines revealed that the estimators were the same 
across the three behaviour categories (p=.19). Attitude 
(regression weight = .007, p=.68) and Intention (regres- 
sion weight = .08, p=.44) did not have a predictive value 
regarding the target behaviour. A significant but small 
effect was observed for the combined factor subjective 
norm/perceived behaviour control (regression weight = 
.09, p=.03). The correct classification rate was therefore 
also low with 55.8%. 

Discussion 
IVIain results 

The present study examined the supposed factorial 
structure and the predictive value of a generic question- 
naire based on theTPB regardlngGPs' wllllgnessto Install 
and use arrlba-llb. In contrast to the findings of other re- 
search In health care [9], [43] we could neither demon- 
strate the TPB's postulated three factorial structure nor 
satisfactorily explain variation In behavioural Intention by 
attitude toward the behaviour, subjective norm and per- 
ceived behaviour control. In our study a two factorial 
model structure combining subjective norm and perceived 
behaviour control had an acceptable model fit even 
though the correlation with attitude was high. 
There might be different reasons for these negative re- 
sults. On the one hand the wording or choice of Items 
used In the questionnaire could have been Inaccurate 
and therefore might have corrupted the factorial structure. 
Nevertheless, as the Included Items were based on sev- 
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eral qualitative studies [7], [27], we assumed them to be 
relevant for our research question. Another reason for 
the results might lie in the choice of the complex interven- 
tions. As arriba-lib included nine different modules, we 
can not exclude that the participating GPs referred to 
different modules when answering the questionnaire. 
Thus, the operationalization of the target behaviour as 
installing and applying arriba-lib might have been too 
general. However, we assume that all modules are of in- 
terest to GPs and can therefore be evaluated in combin- 
ation. Additionally, the use of an electronic decision aid 
as an interactive tool in cooperation with the patient re- 
quires more considerations regarding the presentation 
and communication within the different modules than do 
more structured interventions like guidelines. It might 
also be possible that due to missing skills the use of a 
PC-software imposed retentions or problems to some GPs 
and thus might have influenced the GPs' behaviour in 
different ways. 

Our findings indicate that none of the included predictors 
explain the actual installation and application of arriba- 
lib sufficiently. Neither do attitude and subjective 
norm/perceived behaviour control significantly predict 
intention, nor is intention significantly associated with 
the target behaviour. There was only a correlation of .20 
between intention and behaviour. 
Another reason for the lacking model fit might be the in- 
fluence of external factors like structural pressures on 
the GPs by the health care system. As GPs in Germany 
are primarily paid for patient contacts often minutes this 
might have led to variations in behaviour not connected 
to the TPB. 

The intention-behaviour gap found in our study is a phe- 
nomenon often discussed in matters of the TPB and the 
prediction of behaviour in general [9], [21], [44]. As we 
did not investigate further variables influencing the rela- 
tion between intention and actual behaviour, we can not 
give definitive explanations for this finding. 
Our results suggest that GPs with more positive attitudes, 
stronger subjective norms, and higher perceived beha- 
viour control were more likely to have positive intentions 
to install and apply arriba-lib. Even though regression 
coefficients did not reach significance, attitude was asso- 
ciated most strongly with intention. In contrast to the 
combined factor subjective norm and perceived behaviour 
control, attitude might be regarded as a trait [45], char- 
acterized as a quite stable disposition and difficult to 
change. On the other hand, subjective norm and per- 
ceived behaviour control eventually depend more on 
situational aspects like patients' and colleagues' expect- 
ations or demands of the health care system and can 
thus be classified as state [46]. This assumption can be 
confirmed by the retest reliability, where attitude showed 
a much higher coefficient than the two other variables, 
although these results have to be considered with caution 
because of the relatively high drop-out rate. Further, 
possibly other processes like situation specific emotions 
become relevant for subjective norm and perceived be- 
haviour control. The neglect of emotion in the TPB as 



cognitive attitude-behavioural theory was criticized by 
various researchers [47], [48] and might contribute to 
the missing amount of explained variance. 

Strength and limitations 

The strength of this study lies in the careful development 
of the questionnaire as a tool assessing physicians' will- 
ingness to change professional behaviour in the context 
of the implementation of complex interventions. The 
questionnaire's development followed the recommenda- 
tions as it was theory guided and based on several 
qualitative studies. 

However, in consideration of the negative results it has 
to be discussed how the questionnaire and the study 
design can be improved. 

Although we followed the recommendations [24], [26] 
and based the questionnaire's items on previous qualita- 
tive research, these studies did not refer to a computer- 
ised tool like arriba-lib. Thus, the transferability to other 
complex interventions might be limited. Nevertheless, we 
assume our approach to be adequate as our aim was to 
develop a generic questionnaire, applicable to different 
types of complex interventions. Furthermore we used a 
neutral wording to allow the questionnaire's transfer to 
other interventions. Another limitation concerns the use 
of the 4-point Likert-scale as due to the small variances 
results might have been influenced. Also the operational- 
ization of the intention might have been unfavourable as 
the two items had a different temporal reference point. 
Additionally, as we could not verify the postulated 
factorial structure of our data according to the TPB we 
had to combine subjective norm and perceived behaviour 
control in the ordinal regression. This limits comparison 
to other studies based on a three factorial model of the 
TPB. However, in the light of the large amount of studies 
supporting the assumptions of the TPB and scarcely 
published research not verifying the TPB, a publication 
bias might exist. 

Conclusions 

Based on the results of our study we can conclude that 
the TPB is not an adequate theoretical framework to guide 
the development of a generic questionnaire in the context 
of the implementation of complex interventions. Possibly, 
the actual implementation can not be predicted even by 
health care professionals themselves and their respective 
attitudes and evaluations. Thus, to enable the successful 
implementation of complex medical interventions evalu- 
ators have to go through the whole development and 
evaluation process according to the MRC-model, without 
short cuts. Further, it has to be discussed whether a 
generic instrument is likely to be valid and useful. Regard- 
ing the TPB a potential publication bias supporting the 
theory's assumptions might have to be considered. 
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